Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support to skip schema inference - Archived #43

Closed

Conversation

devendrasr
Copy link

@devendrasr devendrasr commented Mar 7, 2024

Archived this one please refer - #45

The current version does not support complex data type parsing while inferring the schema from within the snapshot.
By the time support for complex data type comes, I am introducing a flag that can be used to skip this flow. This will offload schema parsing to the underlying parquet extension. Here is how you can do it -

scan data:

SELECT * FROM iceberg_scan("s3://my-bucket/icebergwh/someschema/t01", skip_schema_inference = true) limit 10;

scan metadata:

SELECT * FROM iceberg_metadata("s3://my-bucket/icebergwh/someschema/t01", skip_schema_inference = true) limit 10;

scan snapshots:

SELECT * FROM iceberg_snapshots("s3://my-bucket/icebergwh/someschema/t01", skip_schema_inference = true) limit 10;

Note - This PR is built on top of changes requested in the PR - #42

@devendrasr devendrasr changed the title Support to skip schema inference along with reading iceberg tables having gzip compressed metadata files Support to skip schema inference Mar 8, 2024
…ed_metadata

Support to read gzip compressed metadata files
@samansmink
Copy link
Collaborator

hey @devendrasr would you mind rebasing this PR on current main then push? That makes reviewing a little easier here

@devendrasr
Copy link
Author

Closing this pull request and renaming it to archived as I created a fresh and simplified pull request for this feature. @samansmink please review - #45

@devendrasr devendrasr closed this Mar 19, 2024
@devendrasr devendrasr changed the title Support to skip schema inference Support to skip schema inference - Archived Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants